Interacting with 3D Graphic Objects in an Image-Based Environment

This paper presents a novel hybrid rendering system, which allows users to interact with 3D graphic objects in an image-based environment. To represent a complex environment, we use a collection of images captured by a longitudinally aligned camera array, which rotates along a circle. The captured images are indexed and compressed for efficient rendering of the environment. Moreover, users can interactively deploy 3D graphic objects and move around in a constrained 3D cylindrical space in the image-based environment. Unlike traditional virtual reality systems that must take into account all the geometrical objects in the scene, the present 3D graphic rendering engine needs only to deal with a small number of foreground 3D graphic objects in the image-based environment, therefore significantly reducing the 3D graphic processing time. Experiments demonstrate that this system can be used as a new virtual and augmented reality platform for games, virtual tours, virtual real estate, and many other applications. ∗ This work was done while these authors were interns at Microsoft Research, China.